Error Decomposition and Model Complexity

Bayesian information geometry provides a general error decomposition theorem for arbitrary statistical models and a family of information deviations that include Kullback-Leibler information as a special case. When applied to Gaussian measures it takes the classical Hilbert space (Sobolev space) theories for estimation (regression, filtering, approximation, smoothing) as a special case. When the statistical and computational models are properly distinguished, the dilemmas of over-fitting and “curse of dimensionality” disappears, and the optimal model order disregarding computing cost is always infinity.